Statistics Surveys 

Vol. (2012) 1-8 
ISSN: 1935-7516 



Multiple Objects: Error Exponents in 
Hypotheses Testing and Identification 

Evgueni Haroutunian and Parandzem Hakobyan 

Institute for Informatics and Automation Problems 
of the National Academy of Sciences of the Republic of Armenia 

Abstract: We survey a series of investigations of optimal testing of mul- 
tiple hypotheses concerning various multiobject models. 

These studies are a bright instance of application of methods and tech- 
nics developed in Shannon information theory to solution of typical statis- 
tical problems. 

AMS 2000 subject classifications: Primary 60K35, 60K35; secondary 
60K35. 

Keywords and phrases: Multiple hypotheses testing, LAO tests, Many 
independent objects, Dependent objects, Multiobject model, Identification 
of distribution, Testing with rejection of decision, Arbitrarily varying ob- 
ject. 

1. Introduction 

"One can conceive of Information Theory in the broad sence as covering the 
theory of Gaining, Transferring, and Storing Information, where the first is 
usually called Statistics." [2]. 

Shannon information theory and mathematical statistics interaction revealed 
to be effective. This interplay is mutually fruitful, in some works results of 
probability theory and statistics were obtained with application of information- 
theoretical methods and there are studies where statistical results provide ground 
for new findings in information theory [12], [14], [16]-[19], [35], [39], [49], [54], 
[57]-[59]. 

This paper can serve an illustration of application of information-theoretical 
methods in statistics: on one hand this is analogy in problem formulation and 
on the other hand this is employment of technical tools of proof, specifically of 
the method of types [15], [17]. 

It is often necessary in statistical research to make decisions regarding the 
nature and parameters of stochastic model, in particular, the probability distri- 
bution of the object. Decisions can be made on the base of results of observations 
of the object. The vector of results is called a sample. The correspondence be- 
tween samples and hypotheses can be designed based on some selected criterion. 
The procedure of statistical hypotheses detection is called test. 

The classical problem of statistical hypothesis testing refers to two hypothe- 
ses. Based on data samples a statistician makes decision on which of the two 
proposed hypotheses must be accepted. Many mathematical investigations, some 
of which have also applied significance, were implemented in this direction [50]. 
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The need of testing of more than two hypotheses in many scientific and ap- 
plied fields has essentially increased recently. As an instance microarray analysis 
could be mentioned [22]. 

The decisions can be erroneous due to randomness of the sample. The test 
is considered as good if the probabilities of the errors in given conditions are as 
small as possible. 

Frequently the problem was solved for the case of a tests sequence, where 
the probabilities of error decrease exponentially as 2~ NE , when the number of 
observations N tends to the infinity. We call the exponent of error probability 
E the reliability. In case of two hypotheses both reliabilities corresponding to 
two possible error probabilities could not be increased simultaneously, it is an 
accepted way to fix the value of one of the reliabilities and try to make the tests 
sequence get the greatest value of the remaining reliability. Such a test is called 
logarithmically asymptotically optimal (LAO). Such optimal tests were consid- 
ered first by Hoeffding [48], examined later by Csiszar and Longo [18], Tusnady 
[57], [58] (he called such test series exponentially rate optimal (ERO)), Longo 
and Sgarro [52]. The term LAO for testing of two hypotheses was proposed 
by Birgc [11]. Amongst papers on testing, associated with information theory, 
we can also note works of Natarajan [54], Gutman [25], Anantharam [8], Han 
[26] and of many others. Some objectives in this direction were first suggested 
in original introductory article by Dobrushin, Pinskcr and Shiryacv [21]. The 
achievable region of error probability exponents was examined by Tunccl [56] . 

The problem has common features with the issue studied in the information 
theory on interrelation between the rate R of the code and the exponent E of 
the error probability. In information theory the relation E(R) is called according 
to Shannon the reliability function, while R(E) is named the E- capacity, or the 
reliability-rate function, as it was introduced by Haroutunian [28], [34], [43]. 

Simple but actual concept of not only separate but also simultaneous inves- 
tigation of some number of objects of the same type, evidently, was first for- 
mulated by Ahlswede and Haroutunian [6] for reliable testing of distributions 
of multiple items. But simultaneous examination of properties of many similar 
objects may be attractive and effective in plenty of other statistical situations. 

The organization of this paper is as follows. We start with the definitions and 
notations in the next section. In section 3 we introduce the problem of multihy- 
potheses testing concerning one object. In section 4 we consider the reliability 
approach to multihypotheses testing for many independent and dependent ob- 
jects. Section 5 is dedicated to the problem of statistical identification under 
condition of optimality. Section 6 is devoted to description of characteristics of 
LAO hypotheses testing with permission of rejection of decision for the model 
consisting of one and of more independent objects. 

2. Definitions and Notations 

We denote finite sets by script capitals. The cardinality of a set X is denoted as 
\X\. Random variables (RVs), which take values in finite sets X, S are denoted 
by X, S. Probability distributions (PDs) are denoted by Q, P, G, W , V,QoV. 
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Let PD of RV X, characterizing an object, be Q = {Q(x), x € X}, and con- 
ditional PD of RV X for given value of state s of the object be V = {V(x\s), x £ 
X, s e S}. 

The Shannon entropy H Q (X) of RV X with PD Q is: 



A 



The conditional entropy Hpy(X \ S) of RV X for given RV S with correspond- 
ing PDs is: 



H P .y{X\S) = - J2 P(s)V (x\s) log V(x\s). 



A 

xex,ses 



The divergence (Kullback-Leibler information, or "distance" ) of PDs Q and 
G on X is: 

£(Q||G)^]TQ(x)log^, 
xex U(x) 

and conditional divergence of the PD PoV ={P(s)V(x\s), x & X, s E S} and 
PD PoW = {P(s) W{x\s),x e X, se5}is: 



D(PoV\\PoW) =D(V\\W\P) = ^2 P(s)V(x\s) log 



V^a;|s) ' 



For our investigations we use the method of types, one of the important tech- 
nical tools in Shannon theory [17, 15]. The type Q x of a vector x = (xi, xn) £ 
X N is a PD (the empirical distribution) 

where N(x\x) is the number of repetitions of symbol x in vector x. 

The joint type of vectors x G X and s = (si, s 2 , — , sn) £ S N is the PD 

P„ s = {^p^,xex,ses), 

where N(x, s|x, s) is the number of occurrences of symbols pair (x, s) in the pair 
of vectors (x, s). The conditional type of x for given s is conditional PD 

^x|s = {V(x\s),x e X,sG S}, 

defined by relation N(x, s\x, s) = N (x\x)V x \ s (x\s) for all x £ X, s E S. 

We denote by Q N (X) the set of all types of vectors in X N for given N, by 
V N (S) - the set of all types of vectors s in S N and by V N (X\s) - the set of 
all possible conditional types of vectors x in X N for given s E S N . The set 
of vectors x of type Q is denoted by Tq (X) and the family of vectors x of 
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conditional type V for given s G S N of type P by Tj? v (X | s). The set of all 
possible PDs Q on X and PDs P on S is denoted, correspondingly, by Q(X) 
and V{S). 

We need the following frequently used inequalities [17]: 



for any type Q G Q N {X) 

(N + exp{7ViJ Q (A)} <| T${X) \< cMH Q (X)}, (2.3) 

and for any type P G P^(5) and V G V N (X\s) 

(N + 1)-I 5 H*I exp{7Vff P ,y(A|S)} <| T$v(X | s) |< cxp{H P . v (X\S)}. (2.4) 

3. LAO Testing of Multiple Hypotheses for One Object 



The problem of optimal testing of multiple hypotheses was proposed by Do- 
brushin [20], and was investigated in [29]-[33]. Fu and Shen [23] explored the 
case of two hypotheses when side information is absent. The problem concerning 
arbitrarily varying sources solved in [38] was induced by the ideas of the paper 
of Ahlswede [1]. The case of two hypotheses with side information about states 
was considered in [3] . In the same way as in [23] from result on LAO testing, the 
rate-reliability and the reliability-rate functions for arbitrarily varying source 
with side information were obtained in [38] . 

The problem of multiple hypotheses LAO testing for discrete stationary 
Markov source of observations was solved by Haroutunian [30]-[32]. In [37] 
Haroutunian and Grigoryan generalized results from [23], [30]-[32] for multi- 
hypotheses LAO testing by a non-informed statistician for arbitrarily varying 
Markov source. 

Here for clearness we expose the results on multiple hypotheses LAO testing 
for the case of the most simple invariant object. 

Let X be a finite set of values of random variable (RV) X. M possible PDs 
G m = {G m (x), x G X}, m = 1,M, of RV X characterizing the object are 
known. 

The statistician must detect one among M alternative hypotheses G m , using 
sample x = (xi, xn) of results of N independent observations of the object. 

The procedure of decision making is a non-randomized test </?at(x), it can 
be defined by division of the sample space X N on M disjoint subsets = 
{x : </'A r ( x ) — m }i m — 1,M. The set A% consists of all samples x for which 
the hypothesis G m must be adopted. We study the probabilities cti\ m (ifN) of 
the erroneous acceptance of hypothesis Gi provided that G m is true 



I Q N {X) \< (^v + 1) 1 * 1 , 
I v N (x\s) |< + 



(2.1) 



(2.2) 



ai \ m (<p N ) = GZ(A?), l,m=l,M, m + l. 



(3.1) 



imsart-ss ver. 2011/11/15 file: Survey.tex date: March 2, 2013 



E. Haroutunian et al. /Multiple Objects: Hypotheses Testing and Identification 5 



The probability to reject hypothesis G m , when it is true, is also considered 

Ot m \m{^PN) = ^2 OCl\ m {ip N ) 

- G N ('J W ) 

= {i-gZ{aZ)). 

A quadratic matrix of M 2 error probabilities {cti\ m {ip n) , n 



(3.2) 



1,M, I 



1, M} is the power of the tests. 

Error probability exponents of the infinite sequence tp of tests, which we call 
reliabilities, are defined as follows: 

Ei\ m (<p)= lim \ ~ -r? l og ai\ m (ip N ) \ , m,l = l,M. (3.3) 



We see from (3.2) and (3.3) that 



E,, 



udn Ei\ m ((p), m=l,M. 



(3.4) 



The matrix 



/ E x \i ■ ■ ■ Ei\i . . . Em\\ \ 



%) = 



E, 



.E 



M\mi 
Em\M 



\ E 1\M ■ ■ ■ E l\M 

called the reliabilities matrix of the tests sequence ip is the object of our inves- 
tigation. 

We recognize that a sequence <p* of tests is LAO if for given positive values 
of M — 1 diagonal, elements of matrix E(y>*) the procedure provides maximal 
values for all other elements of it. 

Now we form the LAO test by constructing decision sets noted TZm . Given 
strictly positive numbers E m \ m , m = 1,M — 1, we define the following regions: 

A 



n m = {Q : D(Q\\G m ) < E m{m }, m = 1, M - 1, 
K M = {Q: D(Q\\G m ) > E m \ m , m=l,M-l}, 

and corresponding values: 



E* 



-E m \ m {E m \m) E m \ 



m = 1,M- 1, 



(3.5) 

(3.6) 
(3.7) 

(3.8) 



K\i = K\i( E m\ m )= inf D(Q\\Gi), l = l,M, m^l, m = l,M-l, 



(3.9) 
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E* Mlm = E* Mlm {E lll ,E 2l2 ,...,E M _ llM _ 1 )= inf D(Q\\G m ), m = l,M-l, 

r 6 Km 

(3.10) 

^M|M = E M\m( E 1\1> E 2\2,-, E M-1\M-i) = min £ M|m' 

m:m— 1. A'/— 1 



Theorem 3.1 [33]: If for described model all conditional PDs G m , m = 1, M, 

are different in the sense that, D(Gi\\G m ) > 0, I ^ m, and the positive numbers 
E \\ii E 2\ii ■■■> e m-i\m-i are such that the following M — 1 inequalities, called 
compatibility conditions, hold 

E 1{1 < min_£>(G ro ||Gi), 

m=2,M (3-12) 



E m \ m < min 



min EtjE^), min D(Gi\\G n 

,J=l,m-l ;=m+l,L 



2,Af — 1, 



i/ien t/iere exists a LAO sequence ip* of tests, the reliabilities matrix of which 
E (<£>*) = {^^ji;} *s defined in (3.8)-(3.11) and aZZ elements of it are positive. 

When one of inequalities (3.12) is violated, then at least one element of matrix 
E(<p*) is equal to 0. 

The proof of Theorem 3.1 is postponed to the Appendix. 

It is worth to formulate the following useful property of reliabilities matrix 
of the LAO test. 

Remark 3.1 [39]: The diagonal elements of the reliabilities matrix of the 
LAO test in each row are equal only to the element of the last column: 



K\m = E *M\m> and E* m]m < Ef {m , I = 1, M - 1, I ± m, m = 1, M. (3.13) 

That is the elements of the last column are equal to the diagonal elements of 
the same row and due to (3.4) are minimal in this row. Consequently the first 
M — 1 elements of the last column also can play a part as given parameters for 
construction of a LAO test. 



4. Reliability Approach to Multihypotheses Testing for Many 
Objects 

In [6] Ahlswede and Haroutunian proposed a new aspect of the statistical the- 
ory - investigation of models with many objects. This work developed the ideas 
of papers on Information theory [1], [5], of papers on many hypotheses testing 
[29]- [33] and of book [9], devoted to research of sequential procedures solving 
decision problems such as ranking and identification. The problem of hypotheses 
testing for the model consisting of two independent and of two strictly depen- 
dent objects (when they cannot admit the same distribution) with two possible 
hypothetical distributions were solved in [6]. In [39] the specific characteristics 
of the model consisting of K(> 2) objects each independently of others follow- 
ing one of given M(> 2) probability distributions were explored. In [47] the 
model composed by stochastically related objects was investigated. The result 
concerning two independent Markov chains is presented in [36]. In this section 
we expose these results. 
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4-1. Multihypotheses LAO Testing for Many Independent Objects 

Let us now consider the model with three independent similar objects. For 
brevity we solve the problem for three objects, the generalization of the problem 
for K independent objects will be discussed hereafter along the text. 

Let Xi, X2 and X3 be independent RVs taking values in the same finite 
set X, each of them with one of M hypothetical PDs G m = {G m (x), x £ X}. 
These RVs are the characteristics of the objects. The random vector (Xi,X2, X3) 
assumes values (x 1 , x 2 , a; 3 ) <G X 3 . 

Let (xi , X 2, X3) = {(x\, x 2 , x 3 ), ... , (x^, x 2 , x 3 ), ... , (x]y, x 2 N , Xjy)), x\ G X, 
k = 1,3, n = l,N, be a vector of results of N independent observations of the 
family (Xi, X2, X3). The test has to determine unknown PDs of the objects 
on the base of observed data. The detection for each object should be made 
from the same set of hypotheses: G m , rn = 1,M. We call this procedure the 
compound test for three objects and denote it by $n, it can be composed of three 
individual tests ip],f, <p 2 N , <p% for each of three objects. The test (p l N , i = 1,3, 
is a division of the space X N into M disjoint subsets A l m , m = 1,M. The set 
A l m , m = 1,M, contains all vectors Xj for which the hypothesis G m is adopted. 
Hence test $at is realised by division of the space X N x X x X N into M 3 
subsets A mi ,m 2 .m 3 = ■A] ni x Am 2 x -4m 3 , m t = 1> M , i = 1,3. We denote the 
infinite sequence of compound tests by When we have K independent objects 
the test <J> is composed of tests ip 1 , ip 2 , ... , ip K . 

The probability of the falsity of acceptance of hypotheses triple (G/ x , G/ 2 , G/ 3 ) 
by the test <£>at provided that the triple of hypotheses (G mi , G,„ 2 , G r „ 3 ) is true, 
where (m 1 ,m 2 ,m 3 ) ^ (h,h,h), rrk,k = « = 1,3, is: 

a h,l2,h\rn 1 ,m2,m 3 {^N) = G^ ll O G^ l2 O G^ 3 (A^ ^ ^) 

± «) • «) • G* (Af 3 ) 

= £ G^( Xl ) J2 G K X 2) E G ™ 3 ( X 3). 
XiG^tf xjeA" xaGXf 

'1 L 2 ( 3 

The probability to reject a true triple of hypotheses (G mi , G,„ 2 , G„ i3 ) by 
analogy with (3.2) is defined as follows: 

^mi ,m2,rri3 \mi ,rri2 ,m 3 (^iV ) ^ ^ .I2 ,Z 3 |mi ,m2 ,m 3 (^-/v) • (^-1) 

(il ,l2,l3)¥ z i m i,m2,m 3 ) 

We study corresponding reliabilities £i 1 .i 2 ,i 3 | mi ,m 2 .m3 (^) of the sequence of 
tests $, 



£jl,W3|mi,ro 3l m 3 ($) = J im \ ~ T7 lo S a /l ,h ,h l™l ,mj ,m a {®N ) f ! 



m is /i = 1,M, i = 1,3. (4.2) 
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Definitions (4.1) and (4.2) imply (compare with (3.4)) that 

Em 1 ,m2,m 3 \m 1 ,m 2 ,m3 (*£") mm . -^h ,h ,ls\nii ,m 2 ,m 3 {^) ■ (4-3) 

(Il,l2,l3)7 t ( m l,™2,ni3) 

Our aim is to analyze the reliabilities matrix E(<&*) = {£ , ( li ; 2 .; 3 | miim2 . m3 ( ( i ) *)} 
of LAO test sequence for three objects. We call the test sequence LAO for the 
model with many objects if for given positive values of certain part of elements of 
reliabilities matrix the procedure provides maximal values for all other elements 
of it. 

Let us denote by ~E((p z ) the reliabilities matrices of the sequences of tests ip'\ 
i = 1,3. The following Lemma is a generalization of Lemma from [6]. 

Lemma 4.1: If elements Ei\ m (ip l ), m,l = 1,M, i = 1,3, are strictly positive, 
then the following equalities hold for E($), $ = (ip 1 , ip 2 , ip 3 ), 1^, m, = 1, M: 

3 

El lt l 2 ,l 3 \mi,m2,m 3 (^) = ^h\m i ( l P l )j 
i=l 

m, + k, (4.4) 

El u l 2 . l 3 \mi, m 2 ,m 3 ( $ ) = ^ E li\mi{V % ), 
ie [[1,2,3] -fc] 

nrik = h, mi ^ k, k = 1,3, (4.5) 

El lt l 2 ,l 3 \ mi ,m 2t m. 3 ($) = El i \ mi ((p' > ) , 

i = L~3, m k = l k , m t ^ l u k e [[1, 2, 3] - i]. (4.6) 



Equalities (4.4) are valid also if Ei.\ m .(Lp l ) = for several pairs (mi,li) and 
several i. 

The proof of Lemma 4.1 is exposed in Appendix. 

Now we shall show how we can erect the LAO test from the set of com- 
pound tests when 3(M — 1) strictly positive elements of the reliabilities matrix 
EM,M,M\ m ,M,M> ^iW.M.Af \M,m,M and EM,M,M\M,M,m> m = ]-,M — 1, are pre- 
liminarily given. 

The following subset of tests: 

V = {$ : £ ro | m (^) > 0, m=LM, i = L3} 
is distinguished by the property that when $ E T> the elements 

EM,M,M\m,M,M\^)> EM,M,M\M,m,M\^) an d ^M,M,Af|M,Af,m W; J71 = 1, M — 1, 

of the reliabilities matrix are strictly positive. 

Really, because E m \ m (tp l ) > 0, m — 1,M, i = 1,3, then in view of (3.4) 
EMImiv 1 ) are also strictly positive. From equalities (4.4)-(4.6) we obtain that 
the noted elements are strictly positive for $ € V and m = 1, M — 1 

EM,M,M\m,M,M(®) = ^Af|m(V )j (4-7) 
^M,M,M|Af,m,M(^) = E M \ m {(p ), (4.8) 
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E M ,M,M\M,M,m(®) = E M \ m (ip 3 ). (4.9) 

For given positive elements 



EM,M,M\m,M,M> Em,M,M \M,m,M, ^Af.M.Af |M,M,mj TO — 1, M — 1, 

define the following family of decision sets of PDs: 

T^m = {Q : D(Q\\G m ) < E MM ,M\ mi ,mi t m 3 , m i = m , m j = M , * + 3,3 = M} 



m = 1,M - 1, i = 1,3, (4.10) 
= {Q '■ D{Q\\G m ) > E M<M>M \ mi}m2>m3 , m l = m, mj = M, i ^ j,j = 1,3, 



m = 1,M - 1}, i = 1,3. (4.11) 

Define also the elements of the reliability matrix of the compound LAO test 
for three objects: 

E*M, M,M\m,M,M = Em, M, M \m, M, M, 

E *M,M,M\M,m,M = E M ,M,M\M,m,M, (4.12) 
E *M,M,M\M,M,m = E M ,M,M\M,M,m, 
E h.l 2 ,h\ mi .m 2 , mi = ' mi ,.D(Q\\ G m,), 

i = T~3, to* = i fcj mi ^h,i^k,ke [[1, 2, 3] - i], (4.13) 

£ 'mi,m 2 ,'m 3 Ui,i2,m3 = E inf ^ ^WH^W)' 

m k = l k , 771 i y£ l h k = ITS, i G [[1, 2, 3] - ft], (4.14) 

3 

^,(,,1,1,™,^,™. = E inf ( .^(Q||G mi ),m^^i = lT3. (4.15) 



The following theorem is a generalization and improvement of the correspond- 
ing theorem proved in [6] for the case K = 2, M = 2. 

Theorem 4.1 [39]: For considered model with three objects, if all distribu- 
tions G m , m = 1,M, are different, (and equivalently D(Gi\\G m ) > 0, I =/= m, 
l,m= 1,M), then the following statements are valid: 

a) when given strictly positive elements E M M M \. m M M , E M<M>M \ MtmiM an d 
EM,M,M\M,M,m: m = 1, M — 1, meet the following conditions 

max(£/ a/ , M,M\l,M,Mi Em,m,m\m,i,m, Em,m,m\M,m,i) 
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< min D(Gi\\Gi), 

1=2. M 



and for m = 2, M — 1, 
min E, 



EM,M,M\m,M,M < m ^ n 
Eu,M,U\M,m,M < mm 
E M ,M ,M\M ,M .m < m i n 



l.m,m\m,m.rm mill D{Gl\\G m ) 
Z=l,m-1 l=m+l,M 



min E*i,, min J)(G;||G m ) 

l=\,m-\ l=m+l,M 



™5_ £ 4,m,(Km,mi min £>(G,||G m ) 
i=l,m-l i=m+l,M 



(4.16) 

(4.17) 
(4.18) 
(4.19) 



i/ien i/jere exists a LAO test sequence € 2?, i/ie reliability matrix of which 
E(<f>*) is defined in (4.12)-(4.15) and all elements of it are positive, 

b) if even one of the inequalities (4. 16) -(4. 19) is violated, then there exists at 
least one element of the matrix E($*) equal to 0. 

For the proof of Theorem 4.1 see Appendix. 

When we consider the model with K independent objects the generalization 
of Lemma 4.1 will take the following form. 

Lemma 4.2: If elements Ei.^.^tp 1 ), rrii,li = 1,M, i = 1,K, are strictly 
positive, then the following equalities hold for $ = (ip 1 , (p 2 , <p K ): 

K 

£ji,i2,-,iic|mi,m2,...,m K ($) = ^ E h \ mi ((f 1 ), Vfli ^ k, i = 1, K, 

i=l 

El u l 2 ,...,l K \ mi ,m 2 ,...,m K (^) = 2^ E U \„ H (Lp l ). 

For given K(M — 1) strictly positive elements M,...,M\m,M,—M> 
EM,M,...,M\M, m ,...,M, , E M ^ M ,M,\M,M...,m, m = 1, M - 1, for K independent 
objects we can find the LAO test <&* in a way similar to case of three independent 
objects. 

Comment 4.1: Idea to renumber ET-distributions from 1 to M K and con- 
sider them as PDs of one complex object offers an alternative way of testing for 
models with K objects. We can give M K — 1 diagonal elements of corresponding 
large matrix E($) and apply Theorem 3.2 concerning one composite object. In 
this direct algorithm the number of the preliminarily given elements of the ma- 
trix E($) would be greater (because M K - 1 > K(M - 1), M > 2, K > 2) but 
the procedure of calculations would be longer than in our algorithm presented 
above in this section. Our approach to the problem gives also the possibility to 
define the LAO tests for each of the separate objects, but the approach with 
renumbering of /^-vectors of hypotheses does not have this opportunity. In the 
same time in the case of direct algorithm there is opportunity for the investigator 
to define preliminarily the greater number of elements of the matrix E($). 

In applications one of two approaches may be used in conformity with pref- 
erences of the investigator. 

imsart-ss ver. 2011/11/15 file: Survey.tex date: March 2, 2013 



E. Haroutunian et al. /Multiple Objects: Hypotheses Testing and Identification 11 
4-1.1. Example 

Some illustrations of exposed results are in an example concerning two objects. 
The set X = {0, 1} contains two elements and the following PDs are given on 
X: Gi = {0,10; 0,90}, G 2 = {0,85; 0,15}, G 3 = {0,23; 0,77}. As it follows 
from relations (4.12)-(4.15), several elements of the reliability matrix are func- 
tions of one of given elements, there are also elements which are functions of 
two, or three given elements. For example, in Fig. 1 and Fig. 2 the results of 
calculations of functions S 1i2 | 2 ,i(^3,3|i,3j £3,313,2) and £1,212,2(^3,311,3) are pre- 
sented. For these distributions we have min(Z)(G2||Gi), D{G^\\G\)) « 2, 2 and 
min(£ , 2i2 |2.ii ^(G3||G 2 )) ~ 1,4. We see that, when the inequalities (4.16) or 
(4.19) are violated, £1,212,1 = and £1,212,2 = 0. 
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4-2. Multihypotheses LAO Testing for Two Dependent Objects 

We consider characteristics of procedures of LAO testing of probability distribu- 
tions of two related (stochastically, statistically and strictly dependent) objects. 
We use these terms for different kinds of dependence of two objects. 

Let X\ and X2 be RVs taking values in the same finite set X and V(X) be 
the space of all possible distributions on X. 

Let (xi,x 2 ) = ((x\,xf), (x\,X2), ...(x\[,x 2 N )) be a sequence of results of N 
independent observations of the pair of objects. 

First we consider the model, which consists of two stochastically related ob- 
jects. We name so the following more general dependence. There are given Mi 

PDs 

G mi = {G mi (x 1 ), x 1 g X}, mi = 1, Mi. 
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The first object is characterized by RV X\ which has one of these M\ possible 
PDs and the second object is dependent on the first and is characterized by RV 
X2 which can have one of Mi x M 2 conditional PDs 

G m2 \ mi = {Gm2\m 1 {x z \x l ), x x ,x 2 S X}, m 1 = I, Mi, m 2 = 1,M 2 . 

Joint PD of the pair of objects is 

G miiTn2 G mi o G m2 | mi {G mi;m2 (.x , x ), x ,x G X^ , 

where 

Gm^mjti 1 ,! 2 ) ^ G rni (x 1 )G m2 \ mi (x 2 \x 1 ), to x = l,Mi, m 2 = 1,M 2 . 
The probability Gj[ mj (xi, x 2 ) of V-vector (xi,x 2 ) is the following product: 

G'm l! ra 2 (xi,X2) = G^ (xi )G^ 2 | mj (x 2 |xi ) 

A N 

J G mi (.t ti )G„ 12 | mi (x n \x n ), 

n=l 

N N 

with G^(xi) = n G„ ll (.T r 1 l ) and G^ 2ini (x 2 |x x ) = J] G m2 | mi (a£ [a;*). 

n— 1 n— 1 

In somewhat particular case, when Xi and X 2 are related statistically [44], 
[60], the second object depends on index of PD of the first object but not 
depends on value x 1 taken by the first object. The second object is character- 
ized by RV V 2 which can have one of Mi x M 2 conditional PDs G m2 | mi = 
{G m2 \ mi (x 2 ), x 2 e X}, To! = l,Mi, m 2 = 1,M 2 . 

In the third case of strict dependence, the objects X\ and X2 can have only 
different distributions from the same given family of M PDs Gi, G 2 , Gm- 

Discussed in Comment 4.1 the direct approach for LAO testing of PDs for 
two related objects, consisting in consideration of the pair of objects as one 
composite object and then use of Theorem 3.1, is applicable for first two cases 
[45] . But now we consider also another approach. 

Let us remark that test <& N can be composed of a pair of tests <p^ and 
if 2 f° r the separate objects: <fr N = (ip^ip^). Denote by ip 1 , ip 2 and $ the 
infinite sequences of tests for the first, for the second and for the pair of objects, 
respectively. 

Let Xi and X 2 be related stochastically. For the object characterized by Xi 
the non-randomized test (/rj^(xi) can be determined by partition of the sample 
space X N on Mi disjoint subsets Af[ = {xi : ip]^ (xi) = 2i}, Zi = 1, Mi, i.e. the 
set Af consists of vectors Xi for which the PD is adopted. The probability 
a; 1 | mi (^]y) of the erroneous acceptance of PD Gi 1 provided that G mi is true, 
Zi, TOi = 1, Mi, mi h, is defined by the set Af^ 

"hlrmOw) = G^iA*). 
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Wc define the probability to reject G mi , when it is true, as follows 

<V| mi H)= E «li|mi(^) = «m 1 (^0- ( 4 - 2 0) 

Corresponding error probability exponents are: 

Ehlmiiv 1 ) = J im (-TF^S^ilmi^jv-)} > mi,/i = l,Mi. (4.21) 

JV— s-oo JV J 

It follows from (4.20) and (4.21) that 

S mi i TO1 ((^ 1 )= min E^i™^ 1 ), i 1 ,mi = l,Mi. 

For construction of the LAO test we assume given strictly positive numbers 
E mi \ mi , Tn — l,Mx — 1 and wc define regions IZ^, I = 1, Mi as in (3.5)-(3.6). 

For the second object characterized by RV X 2 depending on X\ the non- 
randomized test ipfj(x2, Xi, li), based on vectors (xi,X2) and on the index of 
the hypothesis l\ adopted for Xi, can be given for each l\ and xi by division of 
the sample space X N on M 2 disjoint subsets 



A% lh (xi) = {x 2 : ^(xa.xi.ix) = l 2 }, h = 1,M 1} h = 1,M 2 . (4.22) 
Let 

AZ, 2 = {(x lf x 2 ) : Xl 6 <, x 2 6 A^ lh ( Xl )}. (4.23) 
The probabilities of the erroneous acceptance for (Ji, Z 2 ) 7^ (mi, m 2 ) are 

n — C N ( A N \ 

Corresponding reliabilites are denoted -E; ll ; 2 | mi , m2 and are defined as in (4.2). 
We can upper estimate the probabilities of the erroneous acceptance for (Zi, Z 2 ) 7^ 
(mi,m 2 ) 

^mi ,7712(^^,(2) = E ^mi( X 0^m 2 |roi( X 2| X l) 



E G£>i)G^ |toi (^| {i ( x 0I*0 



< max G^^^^CxOlxO E ^(xi) 

= G^«) max w G^ 2|mi « |ii (x 1 )|x 1 ). 

These upper estimates of ai 1 .i 2 \ mi ,m 2 (^N) for each (Zi,Z 2 ) 7^ (wi,m 2 ) we 
denote by 

A 11 i a | TOll m a (*iv) = 0-0 max G^i^C^i^CxOIxx). 
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Consequently we can deduce that for h,mi = l,Mi, i 2 ,m 2 = 1,M 2 , new 
parameters 

are lower estimates for reliabilities £ ; i 1 .; 2 |„ iljTO2 ($). 

We can introduce for li, mi = 1, M\, Z 2 , m 2 = 1, M 2 , m 2 7^ I2, 

Aalli.mi.ma^) = ma f w G m 2 |m! ( X l) l X l)' 

and also consider 



^malii.mi.ma^jv) = max G m2 \ mi (^m 2 | Zl ( x l) l x l ) 

= E Aalli.mWv*)- (4-24) 

l 2 ^m 2 

The corresponding estimates of the reliabilities of test <p 2 N , are the following 

Fl 2 \l um i, m2 (v 2 ) = Jim^ |--^logA 2 |i 1 ,m 1 ,m 2 (Vw)| , 



h,m 1 = l,Mi, Z 2 ,m 2 = 1,M 2 , m 2 ^ Z 2 . (4.25) 



It is clear from (4.24) and (4.25) that for every h,mi = 1,M±, l 2l m 2 = 1,M 2 

,mi ,7712 

(^ 2 ). (4.26) 

l 2 :l 2 ^m 2 



For given positive numbers •F| 2 |ii,Tn I ,J3> '2 = 1, M 2 — 1, for Q & IZ^ and for each 
pair h,mi = 1, Mi let us define the following regions and values: 



K hlh (Q) = {V : D(V\\G hlh \Q) < F hllumul2 }, l 2 = 1,M 2 - 1, (4.27) 



^jw a |li(0) = {^: D(V\\G hlh \Q) > F hlhtmul2 , i a = l,M 3 -l}, (4.28) 
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A 



Fh\h,mi,h(Fh\h,m u l 3 ) = Fl 3 \h,mi,h, h — l,Af 2 — 1, (4.29) 

K\lum„ m2 (Fl,\lum u l,)= inf inf £>(V||G m2 | mi |Q), 



to 2 = 1,M 2 , ,m 2 ^ fe, Z 2 = 1,M 2 - 1, (4.30) 

^itf 2 |'i,roi,m 2 (-^l|ii,mi,lj — ) ^M 2 -l|'i,TOi,M 2 -l) 



m 2 = l,M 2 -l, (4.31) 

^M 2 |ii,mi,M 2 (-^lKi,mi,l' •••) ^M 2 -l|ii,roi,M 2 -l) 

^ min J£n 1>mi|Ma . (4.32) 
; 2 =i,iw 2 -i 



We denote by F(y> 2 ) the matrix of lower estimates for elements of matrix E(y 2 )- 
Theorem 4.2 [47]: If for given mi,h = l,Mx, all conditional PDs G; 2 |; 17 
li = l,Af 2 , are different in the sense that D(Gi 2 \i 1 \\G m2 \ mi \Q) > 0, for all 
Q € TZ-in h 7^ W2, 77i 2 = 1,M 2 , w/ien i/ie strictly positive numbers i*iu liTOlj i, 
F2\i 1 .m 1 ,2,---,FM 2 -i\ii,m 1 .M 2 -i are such that the following compatibility condi- 
tions hold 

Fm umi ,i < min inf D(G h \ h ||Gi| mi \Q), (4.33) 

l 2 =2,M 2 

Ffa 2 |ii,mi ,m2 

< min min inf D{G l2 \ h \\G m2 \ mi \Q), 

\l 2 =m 2 + l,M 2 Q^ii 
™™— F l 2 \l u m u m 2 ( F h\h,™,x,h) J > ™2 = 1,M 2 - 1, (4.34) 

Z 2 — 1,7772 — 1 / 

£/7e7i i/iere exists a sequence of tests ip 2 '* , such that the lower estimates are 
defined in (4-29)- (4-32) and are strictly positive. 

Inequalities (4.33), (4.34) are necessary for existence of test sequence with 
matrix F(ip 2 >*) of positive lower estimates having given elements Fi 2 \i i mi i 2 , Z 2 = 
1, M 2 — 1 in diagonal. 

Let us define the following subsets of V(X) for given strictly positive elements 
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EMi,l 2 \li,h> F h,M2\h,l 2 > 'l ~~ ~ 1; h — 1,M 2 — 1: 



A 



fcfc = {Q : D(Q||G h ) < S Mlii2 | il)i2 }, h = l.Mx - 1, Z 2 = 1,M 2 - 1, 

(4.35) 



A 



^1^(0) = : DCVHGfai^lQ) < F hM2lhM }, h = l.Mi - Ma = l,M a - 1, 

(4.36) 



A 



ft Ml = {Q : D(Q\\G h ) > E MlMh , h , h = l,M x - 1, l 2 = 1,M 2 - 1}, 

(4.37) 

ftiifcli^Q) = = D(V\\G hlh \Q) > F hMAh , l21 h=XW=l, l 2 = 1,M 2 -1}. 

(4.38) 

Assume also that 

F h,M 2 \h,l 2 = F h,M 2 \h,l 2 , (4.39) 



A 



^m^IMs = ^.iali!,!,. *i = l.Mx - 1, i 2 = 1,M 2 - 1, (4.40) 

KMmuh = WU^W), ( 4 - 41 ) 

^Ih^irf i nf , .i?(^||G TO2/mi |Q), m 2 ^Z 2 , (4.42) 

F hJ 2 \m 1: m 2 ~ F m 1 J 2 \m 1 ,m 2 + F l 1 ,m 2 \m 1 ,m 2 > m i ^ h, * = 1;2, (4-43) 

^rai ,mj Imi ,m 2 ,, , v-?/* 1 , ,'2 ,m 2 • (4.44) 
(li itej^imi ,m 2 ) 



Theorem 4.3 [44]: // aZZ PDs G mi , m 1 = l,Mi, are different, that is 
D(Gi 1 \\G rni ) > 0, li ^ mi, /i,toi = l,Mi, and aZZ conditional PDs G/ 2 |; i; 
Z 2 = 1, M 2 , are a/so different for all l\ = 1, Mi, in i/ie sense that 
D{Gi \i^ ||G m2 | mi |Q) > 0, Z 2 ^ m 2; i/ien i/ie following statements are valid. 

When given strictly positive elements E Ml i 2 \„ lll2 and F ll M2 \i i rll2 , mi = 
1, Mi — 1, m 2 = 1,M 2 — 1, raeei i/ie following conditions 

E Ml ,i 2 \i,i 2 < min £>(g tl HGi), (4.45) 

i!=2,Mi 

F h,M2\h, i < min inf ^(Gi^iJIGi^JQ), (4.46) 

l 2 =2.M 2 Q£><-h 

E Ml ,i 2 \ mu i 2 < min [ min -Ey^u?' min D(G ;i |[G TOl )], 

/i— l,mi — 1 ii= mi+l,Mi 



mi = 2, Mi - 1, (4.47) 
i^M^.ms < min[ min F,* ,„,, „, min inf Z?(G i2 | Zl ||G m2 | mi |Q)], 

Z 2 = l, 7712-1 l 2 =m 2 +l,M 2 



m 2 = 2, M 2 - 1, (4.48) 
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then there exists a LAO test sequence <!>*, the matrix of lower estimates of which 
F($*) = {i J1 ( [ li ; 2 | TOlim2 ($*)} is defined in (4-39)-(4-44) an d a ^ elements of it are 
positive. 

When even one of the inequalities (4-45)-(4-4$) * s violated, then at least one 
element of the lower estimate matrix F($*) is equal to 0. 

When X\ and X 2 are related statistically [44], [60] we will have instead of 
(4.24), (4.25) A% ]h = {x 2 : pf(x a> Zi) =' h}, h = M^i", h = TJT 2 , and 

Af[ t i 3 = {(xi,X2) : xi G Af[, X2 £ Ai^u(xi)}. In that case we have error 
probabilities 

Gm um2 (A^ 1 i 2 ) = (xi)G^ 2 | mi (x 2 ) 

(xi,xa)e^, Ja 

= G ™l( Xl ) G ni 2 \ mi ( X2 ) 

= G^ mi {Ah\h) G mMZl (h,h) + (m 1 ,m 2 ). 

For the second object the conditional probabilities of the erroneous acceptance 
of PD G;,|; 1 provided that G m2 | mi is true, for h,mi = 1,M%, l%,m 2 = 1,M2, 
are the following 

The probability to reject G m ,| mi , when it is true is denoted as follows 

a m 2 \l 1 ,m 1 ,m 2 ('^N) = G m 2 \ mi (-^m 2 \h) = ^ a h\h,mi,m 2 (Vn)' 

h^rn 2 

Thus in the conditions and in the results of Theorems 4.2 and Theorems 4.3 
instead of conditional divergences inf D{Gi 2 \iA\G m2 \ mi \Q), 

inf D(V\\G m2 \ mi \Q) we will have just divergences D(G h \ h \\G m2 \ mi ), 



D{V\\G m 2 \ mi ) and in place of F i2 | il>mi!frt2 ($), F htl2 \ mum2 {§), h,m 1 = l,Mi, 
h,m 2 = l,M2,willbeEj a |j ljmiiWla ($), £i lii2 | miiJn2 ($), h,mi = l,Afi,/ 2 ,m 2 = 
1, M 2 . And in that case regions defined in (4.27), (4.28) will be changed as fol- 
lows: 



a 



Tl l2lh ={V: D(V\\G l2lh ) < E l2llumuh }, Z 2 = 1,M 2 -1, 



A 



^m 2 | ;i -{^: W|G, 2 | Zl )>£;, 2 | iliJniilaJ « 2 = 1,M 2 -1}, 
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In case of two statistically dependent objects the corresponding regions will be 



K h = {Q : D{Q\\G h ) < E MlMh , 2 }, h = 1,M X - 1,Z 2 = M/ 2 - 1, 



A 



n hlh = {V : D(V\\G hlh ) < E luMa a ula }, h = l,Mi - 1, h = l,M a - 1, 



A 



n Ml = {Q ■ D{Q\\G h ) > E MlMluh , h = l,Mj - 1, l 2 = 1,M 2 - 1}, 
K Ua \ h ={V : D(V\\G hlh ) >E luM2lhM , h = MA - 1, Z 2 = 1,M 2 -1}. 

So in this case the matrix of reliabilities E($*) = {-B*( 1) ( 2 | miiTO2 , h,mi 
1, Mi, ?i,mi = l,Mi}, will have the following elements: 

E h.M 2 \h,l 2 = E h,M 2 \h,h> 
E Mi,h\h,l2 = E M 1} l 2 \h,l 2 , 



h = l,Mi -l,h = 1,1* - 1, 

E h,l 2 \mi,m 2 = ^mi J 2 |mi,m 2 ^ii ,m 2 |r«i ,m 2 ' m i ^ h, * = 1.2, 

771* — ' 

m^mjlmi.mj ,, , N m , ln , ii ,i 2 |"U ,m 2 " 
((l,i 2 )/(irii,m 2 ) 



Theorem 4.4: [44] // aZZ PUs G mi , mi = l,Mi, are different, that is 
D{Gi 1 1 \G mi ) > 0, h ^ mi, h,mi = l,Mi, and all conditional PDs G; 2 |; i; l 2 = 
1,M 2) are also different for all h = 1, Mi, in the sense that D{Gi 2 \i 1 ||G m2 | mi ) > 
0, I2 ^ rn2, then the following statements are valid. 



When given strictly positive elements Em-lMIi,!? an< ^ E h,M 2 \h,i 2 i '1 = l)Mi 



I2 = 1, M 2 — 1, meet the following compatibility conditions 
E Ml ,i 2 \i,i 2 < mm_D(Gi 1 \\G 1 ), 

li=2.M l 

Ei u M 2 \h,i< min D{G u \ h \\Gi\ 



l 2 =2,M 2 

EM 1 ,i 2 \m 1 ,i 2 < min 



min Ef ih{rnih , min D(G h \\G m ,) 

ii=l,mi — 1 (i=mi + l,Mi 



mi = 2, il/i - 1 

-Eii,Af 2 |ii,m 2 < min 



min Ei ul3 \i um3 , min D{G U \ U ||G m ,| mi ) 

l 2 = l,m 2 —l l 2 =m 2 + l,M 2 



m 2 = 2,M 2 - 1, 

then there exists a LAO test sequence <!>*, the matrix of which E(<1>*) is stated 
above and all elements of it are positive. 

When even one of the compatibility conditions is violated, then at least one 
element of the matrix E(<£>*) is equal to 0. 
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5. Identification of Distribution for One and for Many Objects 

In [9] Bcchhofcr, Kiefer, and Sobel presented investigations on sequential multiple- 
decision procedures. This book concerns principally with a particular class of 
problems referred to as ranking problems. Chapter 10 of the book by Ahlswcdc 
and Wegener [7] is devoted to statistical identification and ranking. Problems 
of distribution identification and distributions ranking for one object applying 
the concept of optimality developed in [11], [48], [29]- [32] were solved in [6]. In 
papers [40] , [46] , [47] and [53] identification problems for models composed with 
two independent, or strictly dependent objects were investigated. 

In [6] , [40] , [47] and [53] models considered in [9] and [7] and variations of these 
models inspired by the pioneering paper by Ahlswede and Dueck [5], applying 
the concept of optimality developed in [11], [29]-[32], [48], were studied. 

First we formulate the concept of the identification for one object, which was 
considered in [6]. There are known M > 2 possible PDs, related with the object 
in consideration. Identification gives the answer to the question whether r-th 
PD occured, or not. This answer can be given on the base of a sample x and 
by a test (p* N (x). More precisely, identification can be considered as an answer 
to the question: is result I of testing algorithm equal to r (that is I = r), or not 
equal I (that is I ^ r). 

There are two types of error probabilities of identification for each r = 1, M: 
the probability cti^ r \ m=r {ip n) to accept I different from r, when r is in reality, 
and the probability ai =r \ m ^ r {(p jv) that r is accepted by test ippf, when r is not 
correct. 

The probability ai^ r \ m=r ((p jv) coincides with the error probability of testing 
a r | r (( / 9jv) (see (6)) which is equal to a Z|r(¥>iv)- The corresponding reliability 

Ei^ r \ m=r ((p) is equal to E r \ r (<p) which satisfies the equality (3.4). 

And what is the reliability approach to identification? It is necessary to de- 
termine the dependence of optimal reliability E*_, m ^ r upon given E*^ m=r = 
E*i r , which can be assigned a value satisfying conditions analogical to (3.12). 

The result from paper [6] is: 

Theorem 5.1: In the case of distinct hypothetical PDs Gi,G%, ...,Gm, for 
a given sample x we define its type Q, and when Q £ T^i (see (3.5)— (3.7)) we 
accept the hypothesis I. Under condition that the a priori probabilities of all M 
hypotheses are positive the reliability of such identification Ei— r \ m u. r for given 
Eijt r \ m — r = E r \ r is the following: 

Ei =r \ m -L r (E r \ r ) = min inf D(Q\\G m ), r = l,M. 

1 m-.mjtr Q:D(Q\\G T )<E r \ r 

We can accept the supposition of positivity of a priory probabilities of all 
hypotheses with loss of generality, because the PD which is known to have 
probability 0, that is being impossible, must not be included in the studied 
family. 

Now let us consider the model consisting of two independent objects. Let 
hypothetical characteristics of objects X\ and be independent RVs tak- 
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min D(Gi\\G r ), min D(Gi\\G r ) 

i=l,r-l l=r+l,M 



ing values in the same finite set X with one of M PDs. Identification means 
that the statistician has to answer the question whether the pair of distri- 
butions (r\,r 2 ) occurred or not. Now the procedure of testing for two ob- 
jects can be used. Let us study two types of error probabilities for each pair 
(ri.To), n.ra = 1,M. We denote by ok,, , > , , the proba- 

V li zyj •> (h,l 2 ) : F{ri,r 2 )\(mi,m 2 ) = (r 1 ,r 2 ) r 

bility, that pair {r\,T2) is true, but it is rejected. Note that this probability 

is equal to a rur2 \ rur ^ N ). Let a(5 2 ) = ( ri>r2 )| (roi , TO2)7t ( ri , r2 ) be the Probability 
that (ri,r2) is identified, when it is not correct. The corresponding reliabilities 

are ■®(ii,I 3 )^(n,ra)|(mi,m3)=(ri,ra) E r 1 ,r 2 \r 1 ,r 2 &rid -E/(/i ,( 2 )=(ri ,r 2 )| (mi ,m 2 )^(ri ,r 2 ) • 

Our aim is to determine the dependence of ^(ii,2 2 )=(ri,ra)|(rn.i,ma)#(ri,r3) on given 

■^ri.raln.raC^Jv)- 

Let us define for each r, r = 1, M, the following expression: 
-A(r) = n 

Theorem 5.2 [40] :For i/ie model consisting of two independent objects if the 
distributions G m , m = 1,M, are different and the given strictly positive number 
E ri r2 \ ri r ,, satisfy condition 

E rur 2 \r u r 2 <minL4(ri), A(r 2 )], 

then the reliability E/^ i 2 \—r ri r3 \\/ miima \^/ ritf . 2 \ * s defined as follows: 

E (hM = (ri,r 2 )\(m 1 ,m 2 )^(r 1 ,r 2 ) (-^ri,r 3 |ri,r 2 ) 

min [i? mi i ri (^E ri r2 i ri r2 ) , E rri2 i r2 (E ri T2 ri r2 )1 , 

where E mi \ ri (E ritr2 i rii7 . 2 ) and E m2 \ r2 {E ri r2 \ ri r2 ) are determined by (3.9). 

Now we will present the lower estimates of the reliabilities for LAO identi- 
fication for the dependent object which can be then applied for deducing the 
lower estimates of the reliabilities for LAO identification of two related ob- 
jects. There exist two error probabilities for each r 2 = 1,M 2 - the probability 
a h^r 2 \h, mi. m 2 =r 2 (fiw) to accept I2 different from r 2 , when r 2 is in reality, and 
the probability 

a i 2 =r 2 \i 1 ,m 1 .m 2 =£r 2 ( ( p'N) that r 2 is accepted, when it is not correct. 
The upper estimate Pi 2 =i r2 \i 1 . mi ,m 2 =r 2 

(ip 2 N ) for a l2 

=£r 2 \li ,7ni ,m 2 =r 2 

{ip 2 N ) is al- 
ready known, it coincides with fi r2 \i^ mi r2 ( l fi%) which is equal to 
£ A2K1 ,mi ,r-2 ('Pjv ) • The corresponding Fi 2 -£ r2 ^i i mi m2=r2 (tp 2 ) is equal to 

h-hj L r 2 

Fr 2 \i 1 ,m 1 .r 2 { L P 2 )i which satisfies the equality (4.3). 

We determine the optimal dependence of -Pi* =r2 u 1 mi m2 ^r 2 u P on given 

F* 

l 2 =£r 2 \h ,mi,m 2 =r 2 ' 

Theorem 5.3 [47]: In case of distinct PDs Gi\i 1 ,G 2 \i 11 ...,G M2 \i 1 , under 
condition that a priori probabilities of all M 2 hypotheses are strictly positive, for 
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eachr 2 = 1,M 2 the estimate of F l2=r2 \ lumu „ l2 ^ r2 for given Fi 27 tr 2 \h,mi,m 2 =r 2 = 
F r a \h,mi,r 3 is the following: 

F lg=ra\liifni ,Bi2?^a (-^rzl'i ,mi,r 2 ) 

min inf inf D(V\\G m2 \ mi \Q). 

The result of the reliability approach to the problem of identification of the 
probability distributions for two related objects is the following. 

Theorem 5.4: If the distributions G mi and G m2 \ mi , mi — 1,M\, m% = 
1,M2, are different and the given strictly positive number F Tl T2 \ ri r2 satisfies 
the condition 

E r ,\ r , < min min D(G r , \\Gu ), min D(Gu \\G ri ] 

J=l,ri-1 Ji=ri+l,Mi 



or 



F, 



r 2 |!i,mi,r 2 



< nun 



inf min D(G r ,i m , \\Gui, \Q), 
QeTl tl i 2 =l,j- 2 -l 



inf min D(G h \ h \ \G r2 \ m] \Q) 



QeK h i 2 =r 2 +l,M 2 

then the lower estimate ^(ii,; 2 )=(ri,r 2 )|(mi,m 2 )^(n,r'2) of the reliability 
^(ii,i2)=(ri,r 2 )|(mi,m 2 )^(n,r 2 ) can be calculated as follows 

F (h,h) = (ri ,r 2 )\(m\,m 2 )^[r\ ,r 2 ) (-^ri ,r 2 |ri ,r 2 ) 

min I ^ ri | 7ril (-F^ r 2 \r\ ,r 2 )? -^r 2 |Zi ,mi ,m 2 C^ri ,r 2 |ri ,r 2 ) I ) 
mi^ri,m2fr2 

w/iere -E ri | mi (F ri , r2 |n,r 2 ) f r2 |i limi>m2 (^ri,r 2 |n,r 2 ) ar e determined respec- 
tively by (3.9) and (4.30). 

The particular case, when X\ and A~2 are related statistically, was studied in 
[44], [60]. 



6. Multihypotheses Testing With Possibility of Rejection of Decision 

This section is devoted to description of characteristics of LAO hypotheses test- 
ing with permission of decision rejection for the model consisting of one or more 
objects. The multiple hypotheses testing problem with possibility of rejection of 
decision for arbitrarily varying object with side information and for the model of 
two or more independent objects was examined by Haroutunian, Hakobyan and 
Yessayan [41], [42]. These works ware induced by the paper of Nikulin [55] con- 
cerning two hypotheses testing with refusal to take decision. An asymptotically 
optimal classification, in particular hypotheses testing problem with rejection 
of decision ware considered by Gutman [25] . 



imsart-ss ver. 2011/11/15 file: Survey.tex date: March 2, 2013 



E. Haroutunian et al. /Multiple Objects: Hypotheses Testing and Identification 23 



6.1. Many Hypothesis Testing With Rejection of Decision by 
Informed Statistician for Arbitrarily Varying Object 

In this section we consider multiple statistical hypotheses testing with possibility 
of rejecting to make choice between hypotheses concerning distribution of a 
discrete arbitrarily varying object. The arbitrarily varying object is a generalized 
model of the discrete memory less one. Let X be a finite set of values of RV X 1 
and S is an alphabet of states of the object. 

M possible conditional PDs of the characteristic X of the object depending 
on values s of states, are given: 



W m = {W m (x\s), x&X, seS}, m = 1,M,...|5 > 1|, 

but it is not known which of these alternative hypotheses W m , to = 1,M, is 
real PD of the object. The statistician must select one among Al hypotheses, or 
he can withdraw any judgement. It is possible for instance when it is supposed 
that real PD is not in the family of M given PDs. An answer must be given 

using the vector of results of N independent experiments x = (xi, X2, ...£jv) an d 

the vector of states of the object s = (si, s 2 , •••,sjv), s„ £ S, n = 1, N. 

The procedure of decision making is a non-randomized test <piv(x, s), it can 
be defined by division of the sample space X N for each s on M + 1 disjoint 
subsets A%(s) = {x : ip N (x,s) = m}, m = 1, M + 1. The set Af (s), I = 1,M, 
consists of vectors x for which the hypothesis Wi is adopted, and A M+1 (s) 
includes vectors for which the statistician refuses to take a certain answer. 

We study the probabilities of the erroneous acceptance of hypothesis Wi 
provided that W m is true 

an m ((p N ) = max W% MT(s)|s) , m,l = TJT, m ? I. (6.1) 

When decision is declined, but hypothesis W m is true, we consider the following 
probability of error: 

a M +i\m((PN) = max (^m+i( s )I 8 ) ■ 

If the hypothesis W m is true, but it is not accepted, or equivalcntly while the 
statistician accepted one of hypotheses Wi, I = 1, M, I ^ to, or refused to make 
decision, then the probability of error is the following: 

a<m\m(¥>N) = a llm {p N ) = max (^(s)|s) , to = 1, M. (6.2) 
Corresponding reliabilities are defined similarly by to (3.2): 



E l]m (tp) = lim <^ -— \oga llm (^ N ) \, m= 1,M, l = l,M + l. (6.3) 
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It also follows that for every test ip 



The matrix 



E m \ m (f) 



Ei\m(p), rn = 1,M. 



2=1, M+l, l^m 

( E^ . . . Ei\ x ... E M \ 1: E M+1 \ X \ 



(6.4) 



El\m ■ ■ ■ El\m ■ ■ ■ E M \ m , E M+1 \ m 



\ Ey\M • ■ • E[\ M . . . E M \ M E M+1 \ M J 

is the reliabilities matrix of the tests sequence if for the described model. 

We call the test LAO for this model if for given positive values of certain M 
elements of the matrix E(cp) the procedure provides maximal values for other 
elements of it. 

For construction of LAO test positive elements -Eim, Em\m a re supposed 
to be given preliminarily. The optimal dependence of error exponents was deter- 
mined in [41]. This result can be easily generalized for the case of an arbitrarily 
varying Markov source. 



6.2. Multiple Hypotheses LAO Testing With Rejection of Decision 
for Many Independent Objects 

For brevity we consider the problem for two objects, the generalization of the 
problem for K independent objects will be discussed along the text. 

Let Xi and X2 be independent RVs taking values in the same finite set X with 
one of M PDs G m S V{X) 1 m = 1, M. These RVs are the characteristics of the 
corresponding independent objects. The random vector (X\, X2) assumes values 

(x\ar^e X xX. Let (x 1 ^ 2 ) = ((xj.sf), (a£,a£), {x l N ,x 2 N )), x k n e X, 
k = 1, 2, n = 1, N, be a vector of results of N independent observations of the 
pair of RVs (Xi,X2). On the base of observed data the test has to determine 
unknown PDs of the objects or withdraw any judgement. The selection for each 
object should be made from the same set of hypotheses: G m , m = 1,M. We 
call this procedure the compound test for two objects and denote it by $n, 
it can be composed of two individual tests ip]^, tp 2 N for corresponding objects. 
The test (p l N , i = 1,2, can be defined by division of the space X N into M + 1 
disjoint subsets A l m , m = 1, M + 1. The set A l ml m = 1, M contains all vectors 
x 1 for which the hypothesis G m is adopted and A l M+1 includes all vectors for 
which the test refuses to take a certain answer. Hence $tv is division of the 
space X N x X N into (M + l) 2 subsets A mi ,m 2 = A] ni x A 2 m2 , m t = 1,M + 1. 
We again denote the infinite sequences of tests by <£, if 1 , (p 2 . 

Let ai lj i 2 | mij m2 (^n) be the probability of the erroneous acceptance of the pair 
of hypotheses (G; 1 ,G; 2 ) by the test &n provided that the pair of hypotheses 
(G mi ,G m2 ) is true, where (mi,m 2 ) ^ (h,h), m% = 1,M, U = l,M, i = 1,2: 

a Ji,J2|mi,ma(^v) = G mi o G„ l2 {Al Xt l 3 ) 
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When the pair of hypotheses (G mi ,G m2 ), mi,m 2 = 1,M is true, but we 
decline the decision the corresponding probabilities of errors are: 

^M+l,M+l|mi ,m2 

($jv) — G mi o G m2 (AM+i,M+i) 

= ^roi (-^M+l) ' G^ n2 {A\ [+1 ). 

or 

a M+l,l 2 \m 1 ,m 2 (^N) = G mi (A\ I+1 ) ■ G m2 (Af 2 ) 

or 

ai u M+i\ mi ,m 2 (^N) = G mi (AlJ ■ G^ ri2 {A 2 M+1 ). 

If the pair of hypotheses (G mi , G m2 ) is true, but it is not accepted, or cquiv- 
alently while the statistician accepted one of hypotheses (G^jG^), or refused 
to make decision, then the probability of error is the following: 

^mi ,7712 \m\ ,7712 (*jv), (6.5) 

k = l,M + l, nii = 1, M, i = T72. 
We study reliabilities £ , i 1 ,i 2 |mi,m 2 ( ( J > ) °f the sequence of tests $, 



Ei u i 2 \ mi . m2 {^) = Jim - — loga ilii2 | miim2 ($jv), (6.6) 

N^-OO iV 

mj,= 1,M, /i = 1,M+ 1, i = T72. 
Definitions (6.5) and (6.6) imply that 

E mi , m2 \mi,m 2 (®) = „ , mm £7j liJa | miim2 ($). (6.7) 

(il,l2)#(mi,"i2) 

We can erect the LAO test from the set of compound tests when 2M strictly 
positive elements of the reliability matrix i?A/+i.m|m.m an d E m M+i\ m ,m, m = 
1,M, are preliminarily given (see [41]). 

Remark 6.1: It is necessary to note that the problem of reliabilities inves- 
tigation for LAO testing of many hypotheses with possibility of rejection of 
decision for the model consisting of two or more independent objects can not 
be solved by the direct method of renumbering. 

7. Conclusion and Open Problems 

"A broad class of statistical problems arises in the framework of hypothesis 
testing in the spirit of identification for different kinds of sources, with complete 
or partial side information or without it. Paper [6] is a start." [2]. 

In this paper, we exposed solutions of a part of possible problems concerning 
algorithms of distributions optimal testing for certain classes of one, or multiple 
objects. For the same models PD optimal identification is discussed again in the 
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spirit of error probability exponents optimal dependence. But these investiga- 
tions can be continued in plenty directions. 

Some problems formulated in [6] and [43] , particularly, concerning the remote 
statistical inference formulated by Berger [10], examined in part by Ahlwede and 
Csiszar [4] and Han and Amari [27] still rest open. 

All our results concern with discrete distribution, it is necessary to study 
many objects with general distributions as in [26]. For multiple objects mul- 
tistage and sequential testing [13] can be also considered. Problems for many 
objects are present in statistics with fuzzy data [24], bayessian detection of 
multiple hypotheses testing [51] and geometric interpretations of tests [61]. 



8. Appendix 

Proof of Theorem 3.1: Probability G^(x) for x G Tq (X) can be presented as 
follows: 

N 
n=l 

X 

= U G ^) NQ{X)) 

X 



Q(x)\ogQ{x) 



G m (x) 

= exp {-N[D(Q || G m ) + H Q (X)}}. (8.1) 
Let us consider the sequence of tests yjv(x) defined by the sets 

B%P= (J Tq(X), m=\JA. (8.2) 
Each x is in one and only in one of B^ , that is 

M 



B^fie^C Mm, and |J flW = X« 

m—l 



Really, for I = 1, A I — 2, m = 2, A I — 1, for each Z < m let us consider arbitrary 
x 6 B\ N) . It follows from (3.5) and (3.7) that there exists type Q G Q N (X) 
such that D(Q\\Gi) < and x G Tq(X). From (3.12) and (3.9) we have 

E m \ m < E*, m (Ei\i) < D(Q\\G m ). From definition of Bm ) we sec that x ^ B^\ 
Definitions (3.10), (3.12) and (3.7) show also that 
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Now, let us remark that for m = l,M — 1, using (2.1), (2.3), (3.1)-(3.3) and 
(8.1) we can estimate orJV ((p*) as follows: 



= <#( [J T${X) 

K Q:D(Q\\G m )>E mlm 



<(N + 1)1*1 sup G m (Tg(X)) 

Q:D(Q\\G m )>E m , m 



\X\ 

<(N + 1)1*1 sup exp{-JVD(Q||G m )} 

Q:U(Q||G ra )>B m | w> 

<exp(-iV[ inf D(Q||G m )- 0JV (l)]} 
< exp{-JV[S m | m -ojv(1)]} , 



where ojy(l) — » with iV — > oo. 

For Z = 1,M - 1, m = T^M, Z ^ m, using (2.1), (2.3), (3.1)-(3.3) and (8.1), 
we can obtain similar estimates: 

a llm {<p* N )=G% (b^) 

= Gl[ (J T Q N (X)) 

\Q :D(Q\\Gi)Em j 

<(N + 1)1*1 sup G»{T${X) 

Q:£>(Q||G m )<B,|, 

<(iV + 1)1*1 sup exp{-7V£>(Q||G m )} 

Q:D(Q\\G m )<E m 

= exp(-7vf inf D(Q\\G m ) - o N (l) U. (8.3) 
1 V<3 : - D (QIIG m )<s i | i 7 J 
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Now let us prove the inverse inequality: 



= G N m [ (J T${X) 

\Q:D(Q\\Gi)<En, 

> sup GZ(T Q (X) 

Q:D(Q\\Gi)<Em 

>(iV+l)-^i sup exp{-ND(Q\\G m )} 

Q:D(Q\\Gi)<Eu, 



According to the definition (3.3) En m (ip*) = lim {— N 1 log ai\ m ((p%)\, tak- 

JV— >oo 

ing into account (8.3), (8.4) and the continuity of the functional D(Q\\Gi) we 
obtain that 

lim {— N^ 1 log a m u(tf* N )\ exists and in correspondence with (3.9) equals to 

TV— >oo 

E*^ { . Thus Ei\ m ((p*) = E*, m , m = 1, M, I = 1, M — 1, I ^ m . Similarly we can 

obtain upper and lower bounds for aM|m(¥>jv)> m = l,M. Applying the same 
resonnement we get that the reliability EM\ m (tp*) = E* M ^ m . By the definition 
(3.4) E m \m(<p*) = E* M \ M . The proof of the first part of the theorem will be 
accomplished if we demonstrate that the sequence of tests ip* is LAO, that is 
for given E^, Em-i\m—i an d every sequence of tests ip for all l,m G 1,-M, 
E mll (<p) < E* mjl . 

Let us consider any other sequence ip** of tests which are defined by the sets 
V[ N \ ...,V ( A f ] such that 



Enm&n > Et ]m , m,/ = l,M. (8.5) 
These conditions are equivalent for N large enough to the inequalities 



ai\m(v*N) < ai\ m {tp* N ), m,l = l,M. (8.6) 

Let us examine the sets Dm C\Bm \ rn = 1,M. This intersection cannot be 
empty, because in that case 

> Gl (B^) 

> max GZ {T£(X)) 

Q:D(Q\\G m )<E m{m V W ' 

> exp{-N(E rn[m + o N (l))}. 
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Let us show that T>\ N ' f] Bm = 0> m , I = 1,M — 1, m ^ I. If there exists Q 
such that D(Q\\G m ) < E m]m and T$(X) £ v\ N \ then 

ai]m (VN) = G% (P ( (Ar) ) 

> exp{-N{E m]m + Oiv(l))}. 
When ^ f)T$ (X) ^T^(X), we also obtain that 

a l]m (<p%) = GZ{V\ N) 

> GZ(vrf]r Q N (x) 

> exp{-N(E m \ m + ojv(l))}. 

Thus it follows that £?;i m (yj**) < E m i m , which in turn according to (3.4) provides 
that Ei\ m {<p**) = E m \ m . From condition (3.12) it follows that E m \ m < E*, m , for 
all I = l,m — 1, hence Ei\ m (ip**) < E*, for alii = l,m — 1, which contradicts 

to (8.5). Hence we obtain that V ( ,n ] fl ®™ ] = B I ( ,f ) for m = l,M - 1. The 
intersection f] B^ is empty too, because otherwise 

Q!M|m(Vjv) ^ «A/|m(^Jv): 

which contradicts to (8.6), hence Vm^ = Bm\ m = 1,M. 

The proof of the second part of the Theorem 3.1 is simple. If one of the 

conditions (3.12) is violated, then from (3.9)-(3.11) it follows that at least one 

of the elements E m u is equal to 0. For example, let in (3.12) the m-th condition 

be violated. It min D(Gi\\G m ), then there exists I* £ 

l=m+i,M 

m+ 1,M such that E m \ m > D(G*\\G m ). From latter and (3.9) we obtain that 

Km = °- 

The theorem is proved. 

Proof of Lemma J^.l: It follows from the independence of the objects that 

3 

mi ,7712 ,777,3 

i=i 

aZi,W3|mi,m 2 ,m 3 (*iv) = i 1 - a h\m k (<PN)) Y[ 

i£[[l,2,3]-fc] 

rrife = Ik, mi ^ k, k = 1, 3, i ^ k, (8.8) 

ai 1 ,l 2> l 3 \ mi>m2 , m3 {§N)=a h \ mi {ip t N ) Y[ i 1 ~ a h\m k (<PN)) i 

fcG[[l,2,3]-i] 

TOfc = Zfc, TTtj ^ i = 1,3. (8.9) 
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Remark that here we consider also the probabilities of right (not erroneous) 
decisions. Because Ei\ m (tp l ) are strictly positive then the error probability 
cni\ m ((p l N ) tends to zero, when TV — > oo. According this fact we have 

T / 1 1 ft ( i \\\ T «i|m (yjy) log (1 ~ ai\m(<PN)) 

= 0. (8.10) 



From definitions (4.2), equalities (8.7)-(8.9), applying (8.10) we obtain relations 
(4.4M4.6). 

The Lemma is proved. 

Proof of Theorem 4.1: The test $* = (i/<*, v? 2 **, <^ 3 >*), where ip 1 '* , i = 
are LAO tests of objects Xi, belongs to the set V. Our aim is to prove that such 
$* is a compound LAO test. Conditions (4.16)-(4.19) imply that inequalities 
analogous to (3.12) hold simultaneously for tests for three separate objects. 

Let the test $ e V be such that E MiMM \ nMiM ($) = E M<M>M \ m ,M,M 

EM,M,M\M,m,Al(^) = ^M,M,M\M,m,M, an d EM,M,M\m,M,M($) = E^[_ M,M\M,M,m> 
m = 1,M - 1. 

Taking into account (4.7)-(4.9) we can see that conditions (4.16)-(4.19) for 
every m = 1, M — 1 may be replaced by the following inequalities: 

E M \ m {^ 1 ) <tooi min inf D{Q\\Gi), min_£>(G;||G m ) . 

1=1^=1 Q:D(Q\\G m )<E M \ m (<p<-) l=m+l,M 

_ (8.11) 
According to Remark 3.1 for LAO test (f 1 '*, i = 1,3, we obtain that (8.11) 
meets conditions (3.12) of Theorem 3.1 for each test $ e D, E m \ m (ip l ) > 0, 
i = 1,3, hence it follows from (3.3) that E m \i((p z ) are also strictly positive. Thus 
for a test $ £ V conditions of Lemma 4.1 are fulfilled and the elements of the 
reliability matrix E($) coincide with elements of matrix E(</j l ), i = 1,3, or sums 
of them. Then from definition of LAO test it follows that Ei\ m ((p % ) < Ei\ m (ip 1 '*), 
then E lul2>l3 \ mitm2im3 ($) < E luhils \ mum2i7n3 ($*). Consequently $* is a LAO 
test and E hMMm ^ m2 , m3 {<5>*) verify (4.12)-(4.15). 

b) When even one of the inequalities (4.16)-(4.19) is violated, then at least 
one of inequalities (8.11) is violated. Then from Theorem 3.2 one of elements 
E m \i(ifi 1 '*) is equal to zero. Suppose E^if 1 '*) — 0, then the elements 

E 3 , m ,l\2,m,l(®*) = EspiV 1 '*) = 0- 

The Theorem is proved. 
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